On Latent Variable Models for Bayesian Inference with Stable Distributions and Processes

Extreme values and skewness in time-series are often observed in engineering, financial and biological applications. This thesis is a study motivated by the need of efficient and reliable Bayesian inference methods when the $\alpha$-stable model is selected to represent such data. The class of stable distributions is the limit of the generalized central limit theorem (CLT), having a key role in representing phenomena that can be thought of as the sum of many perturbations, with potentially unbounded variance. Besides the ability to model heavy-tailedness, another consequence of the generalized CLT is a further degree of freedom of stable distributions, namely their potential skewness. However, stable distributions are, at the same time, highly intractable for inference purposes. Several approximate methods are available in the literature, in both the frequentist and Bayesian paradigms, but they suffer from a number of deficiencies, the greatest of which is the lack of quantification of the approximation in place. This thesis proposes Bayesian inference schemes for two different latent variable models, with the aim of providing guarantees of accuracy when the $\alpha$-stable model is used. In the first part of the thesis, a marginal representation of the $\alpha$-stable density is used to develop a novel, asymptotically exact, Bayesian method for parameter inference. This is based on the pseudo-marginal Markov chain Monte Carlo (MCMC) approach, that requires only unbiased estimates of the intractable likelihood, computed through adaptive importance sampling for the marginal representation. The results obtained are comparable to a state of the art conditional Gibbs sampler, but do not introduce any approximation, while allowing for better control of the quality of the inference. The focus of the second and central part of the thesis is the Poisson series representation (PSR) of $\alpha$-stable random variables. An approach that turns the infinite-dimensional PSR into an approximately conditionally Gaussian representation, by means of Gaussian approximation of the residual of the series, has been presented in previous literature, together with inference procedures such as MCMC and Particle Filtering. In this setting, the first contribution of this dissertation is the formulation of a CLT for the PSR residual, which serves to justify the existing approximation. Moreover, numerical and theoretical results on the rate of convergence for finite values of the truncation parameter are presented. The convergence is examined directly in terms of Kolmogorov distance between distribution functions, through the application of probability theoretic results, such as the Ess$\acute e$en’s smoothing lemma. This analysis allows for the selection of appropriate truncations for different $\alpha$-stable parameter configurations and gives theoretical guarantees on the accuracy achieved when using the PSR model. Furthermore, superior behaviour of the proposed approximation is found, compared to the simple series truncation, justifying its use for inference tasks. In the third and final part of this thesis, an extension of the modified Poisson series representation (MPSR) of linear continuous-time models driven by $\alpha$-stable L$\acute e$vy processes to the multivariate case is presented. Stable L$\acute e$vy processes are suitable to model jumps and discontinuities in the state, while possessing the self-similarity property, which makes these processes a very natural class for the driving noise in continuous time models. A scheme for approximate simulation from the multivariate linear models, namely multivariate stable vectors evolving in time, is presented. While stable random vectors are parametrized by a function, the presented approximate approach involves only finite dimensional parameters. This will facilitate inference methods, to be developed in future work, towards which the proposed simulation methods constitute the foundational work

Optimal Quantisation of Probability Measures Using Maximum Mean Discrepancy

Several researchers have proposed minimisation of maximum mean discrepancy (MMD) as a method to quantise probability measures, i.e., to approximate a distribution by a representative point set. We consider sequential algorithms that greedily minimise MMD over a discrete candidate set. We propose a novel non-myopic algorithm and, in order to both improve statistical efficiency and reduce computational cost, we investigate a variant that applies this technique to a mini-batch of the candidate set at each iteration. When the candidate points are sampled from the target, the consistency of these new algorithms—and their mini-batch variants—is established. We demonstrate the algorithms on a range of important computational problems, including optimisation of nodes in Bayesian cubature and the thinning of Markov chain output

Sharp Gaussian Approximation Bounds for Linear Systems with α-stable Noise

We report the results of several theoretical studies into the convergence rate for certain random series representations of α-stable random variables, which are motivated by and find application in modelling heavy-tailed noise in time series analysis, inference, and stochastic processes. The use of α-stable noise distributions generally leads to analytically intractable inference problems. The particular version of the Poisson series representation invoked here implies that the resulting distributions are “conditionally Gaussian,” for which inference is relatively straightforward, although an infinite series is still involved. Our approach is to approximate the residual (or “tail”) part of the series from some point, c > 0, say, to ∞, as a Gaussian random variable. Empirically, this approximation has been found to be very accurate for large c. We study the rate of convergence, as c → ∞, of this Gaussian approximation. This allows the selection of appropriate truncation parameters, so that a desired level of accuracy for the approximate model can be achieved. Explicit, nonasymptotic bounds are obtained for the Kolmogorov distance between the relevant distribution functions, through the application of probability-theoretic tools. The theoretical results obtained are found to be in very close agreement with numerical results obtained in earlier work

Optimal Thinning of MCMC Output

The use of heuristics to assess the convergence and compress the output of Markov chain Monte Carlo can be sub-optimal in terms of the empirical approximations that are produced. Typically a number of the initial states are attributed to "burn in" and removed, whilst the remainder of the chain is "thinned" if compression is also required. In this paper we consider the problem of retrospectively selecting a subset of states, of fixed cardinality, from the sample path such that the approximation provided by their empirical distribution is close to optimal. A novel method is proposed, based on greedy minimisation of a kernel Stein discrepancy, that is suitable for problems where heavy compression is required. Theoretical results guarantee consistency of the method and its effectiveness is demonstrated in the challenging context of parameter inference for ordinary differential equations. Software is available in the Stein Thinning package in Python, R and MATLAB.Comment: To appear in the Journal of the Royal Statistical Society, Series B, 2021

Nonasymptotic Gaussian Approximation for Inference with Stable Noise

The results of a series of theoretical studies are reported, examining the convergence rate for different approximate representations of $\alpha$-stable distributions. Although they play a key role in modelling random processes with jumps and discontinuities, the use of $\alpha$-stable distributions in inference often leads to analytically intractable problems. The LePage series, which is a probabilistic representation employed in this work, is used to transform an intractable, infinite-dimensional inference problem into a conditionally Gaussian parametric problem. A major component of our approach is the approximation of the tail of this series by a Gaussian random variable. Standard statistical techniques, such as Expectation-Maximization, Markov chain Monte Carlo, and Particle Filtering, can then be applied. In addition to the asymptotic normality of the tail of this series, we establish explicit, nonasymptotic bounds on the approximation error. Their proofs follow classical Fourier-analytic arguments, using Ess\'{e}en's smoothing lemma. Specifically, we consider the distance between the distributions of: $(i)$~the tail of the series and an appropriate Gaussian; $(ii)$~the full series and the truncated series; and $(iii)$~the full series and the truncated series with an added Gaussian term. In all three cases, sharp bounds are established, and the theoretical results are compared with the actual distances (computed numerically) in specific examples of symmetric $\alpha$-stable distributions. This analysis facilitates the selection of appropriate truncations in practice and offers theoretical guarantees for the accuracy of resulting estimates. One of the main conclusions obtained is that, for the purposes of inference, the use of a truncated series together with an approximately Gaussian error term has superior statistical properties and is likely a preferable choice in practice.Comment: V1: 41 pages, 16 figures. V2: Text typos fixed; redundant figures from main text and appendices removed; added references in section I; changed section VI and its proofs in Appendices C-D-E; improved section IX; removed section X (Discussion and Conclusion), reference style changed. V3: title in the metadata updated. V4: Updated Theorem 1 and its proof, global revisio